Building a Continuous Integration Workflow

Continuous integration automation tasks include cloning the repository at a specific commit, linting, building, and testing code, and evaluating changes to test coverage. The goal of continuous integration automation is to provide a guard against code changes that will lower the quality of a project or violate the rules codified in automation.

In this lesson, we will learn how to create a continuous integration workflow. In our continuous integration workflow, we will learn to execute jobs across multiple operating systems concurrently. We will install build tools onto the job executors, which we will use to build the software project. We will clone the source code for the project using an action. Finally, we will enforce passing tests and code quality by running a code linter and executing unit tests.

Introducing the tweeter command-line tool#

We cannot have a continuous integration workflow without a software project to run the workflow upon. We will be using a simple Go command-line tool called tweeter. The source code for the project can be found here on GitHub.

Tweeter is a simple Go command-line tool that will send tweets to Twitter. The source code consists of two packages, main and tweeter. The tweeter package contains Go tests that will be executed by our continuous integration workflow.

Cloning and testing tweeter#

Create a new repository from this template by clicking the “Use this template” button in the repository. This is shown below:

This will create a copy of the repository in our account. Run the following commands to clone and test tweeter (replace your-account with your account name and repository-name with the name of the cloned repository):

These commands can be seen executed in the terminal below.

Executing tweeter with the -h argument will provide usage documentation. Test it out by entering the following in the terminal above:

If we are not inclined to use social media, tweeter also allows users to simulate sending a tweet. When -–dryRun is specified, the message value will be output to STDOUT, rather than being sent to Twitter as a tweet.

Test it out in the terminal above with the following command:

Next, we will build a continuous integration workflow to test tweeter.

Goals of the tweeter continuous integration workflow#

Before building a continuous integration workflow, we should consider what we want to accomplish with the workflow. For the tweeter workflow, our goals are the following:

• Trigger on pushes to main and tags formatted as a semantic version – for example, v1.2.3 must build and validate.

• Pull requests against the main branch must build and validate.

• Tweeter must build and validate on Ubuntu, macOS, and Windows concurrently.

• Tweeter must build and validate using Go 1.16 and 1.17 concurrently.

• Tweeter source code must pass a code-linting quality check.

Continuous integration workflow for tweeter#

With our goals for the tweeter continuous integration workflow specified, we can construct a workflow to achieve those goals. The following is a continuous integration workflow that achieves each goal:

The preceding workflow is a lot to absorb initially. However, if we break down the workflow, the behavior will become clear.

Triggering the workflow#

The first two goals for the tweeter continuous integration workflow are as follows:

• Pushes to main and tags matching v[0-9]+.[0-9]+.* must build and validate.

• Pull requests against the main branch must build and validate.

These goals are accomplished by specifying the following event triggers:

The push: trigger will execute the workflow if a tag is pushed matching v[0-9]+.[0-9]+.* – for example, v1.2.3 would match the pattern. The push: trigger will also execute the workflow if a commit is pushed to main. The pull_request trigger will execute the workflow on any changes to a pull request targeting the main branch.

Note that using the pull_request trigger will allow us to update the workflow and see the changes to the workflow each time the changes are pushed in a pull request. This is the desired behavior when developing a workflow, but it does open automation to malicious actors. For example, a malicious actor can open a new pull request, mutating the workflow to exfiltrate secrets exposed in it. There are multiple mitigations to prevent this, which can be applied independently or together, depending on the security preferences of a given project:

• Only allow maintainers to trigger workflows.

• Use the pull_request_target event to trigger, which will use workflows defined in the base of the pull request without regard to workflow changes in the pull request.

• Add a label guard for executing a workflow so that it will only execute if a maintainer adds the label to the pull request. For example, a pull request can be reviewed by a maintainer, and then if the user and code changes are safe, the maintainer will apply a safe-to-test label, allowing the job to proceed.

Next, we'll extend automation to include multiple platforms and Go versions.

Entering the matrix#

The next two goals for the tweeter continuous integration workflow are as follows:

• Tweeter must build and validate on Ubuntu, macOS, and Windows concurrently.

• Tweeter must build and validate using Go 1.16 and 1.17 concurrently.

These goals are accomplished by specifying the following matrix configuration:

The test job specifies a matrix strategy with two dimensions, go-version and os. There are two Go versions and three OSs specified. This variable combination will create six concurrent jobs, [(ubuntu-latest, 1.16.x), (ubuntu-latest, 1.17.x), (macos-latest, 1.16.x), (macos-latest, 1.17.x), (windows-latest, 1.16.x), and (windows-latest, 1.17.x)]. The values of the matrix will be substituted in runs-on: and go-version: to execute a concurrent job, satisfying the goals of running on each combination of platform and Go version:

In the preceding figure, we can see each matrix job executing concurrently. Note that each job specifies the name of the job, test, and the matrix variables for the job.

Building, testing, and linting#

There is an overlap of build, testing, and linting in the last three goals:

• Tweeter must build and validate on Ubuntu, macOS, and Windows concurrently.

• Tweeter must build and validate using Go 1.16 and 1.17 concurrently.

• The Tweeter source code must pass a code-linting quality check.

The following steps will satisfy these requirements:

In the preceding steps, the following occurs:

1. Go is installed with the actions/setup-go@v2 action using the matrixspecified Go version. This action is available to all GitHub users and is published through the GitHub Marketplace. There are numerous actions available in the Marketplace that can simplify workflow authoring.

2. The source code for the current ref is cloned with the actions/checkout@v2 action in the current working directory. Note that the action is not named. For commonly used actions, it is idiomatic to not provide a name.

3. Linting is run with the golangci/golangci-lint-action@v2, which installs and executes the golangci-lint tool on the source of the repository, satisfying the goal of ensuring that the code passes a lint quality check. This particular action includes several sub-linters that run a rigorous check of common Go performance and stylistic errors.

4. The code is functionally validated by running an ad hoc go test ./... script, which tests the packages recursively in the repository. Note that in a previous step, the Go tools have been installed and are available for use in subsequent steps.

With the preceding steps, we have satisfied the goals of our continuous integration workflow. With the preceding workflow, we executed a matrix of concurrent jobs, installed build tools, cloned source code, linted, and tested the change set. In this example, we learned to build a continuous integration workflow for a Go project, but any language and set of tools can be used to create a continuous integration workflow.